Where language models go blind

1. One fact, and what follows from it

A large language model is trained to predict the next token in a sequence of language, and that training objective is the key to both its power and its limits. Where a task really is sequence-modelling over language — summarising, paraphrasing, drafting, translating — the model's prior is exactly the right one. Where the task's true object is something else, the same prior is not merely weaker; it is aimed at the wrong target.

This reframing matters because it changes what counts as a fix. If a model gets an arithmetic problem wrong, the instinct is to prompt it more carefully or to use a larger model. But arbitrary-precision arithmetic is not an emergent property of better next-token prediction; it is a different kind of computation. No prompt makes a language model into a calculator. The productive question is not “how do we make the model do this?” but “what mechanism should do this, and how does the model hand off to it?”

2. A map of the blind spots

Four regions recur across every current model. They differ in mechanism but share the same root: the native object is not a linguistic sequence.

Two-dimensional and spatial structure

A table is two-dimensional; the model reads a one-dimensional stream. Serialised into a line, a wide spreadsheet loses the correspondence between a value and its column, and the model's answers about it degrade as the table widens (Wu et al., 2025). The same blindness applies to fine spatial geometry: a model cannot triangulate over a dense set of raw coordinates it has been handed as text.

Exact numbers

Numbers are tokenised into fragments, and multi-digit arithmetic requires carries to propagate across those fragments in a way the architecture does not reliably support. Long decimals and scientific notation shred into meaningless pieces. The failure is quiet: the model returns a confident figure that is simply wrong.

Deep compositional logic

On problems whose difficulty grows by composition — more constraints, more steps, deeper nesting — accuracy does not decline gracefully. It holds, then collapses past a threshold (Shojaee et al., 2025). This is the region most often mistaken for a general verdict about “reasoning”, and §3 takes it apart.

Self-verification

A single forward pass emits tokens one at a time with no built-in check; the model cannot see its own error until the sequence is already written. Reasoning-style models and external verifiers add checks around a call, but a naive single call has none.

3. One failure that is really two

Careful sourcing matters most where a claim has travelled widely. In 2024, a study introduced controlled perturbations of grade-school maths problems and reported that adding a single irrelevant (“no-op”) clause could reduce a model's accuracy by as much as 65% — presented as evidence that models rely on pattern-matching rather than reasoning (Mirzadeh et al., 2025). The result was influential and is still cited as though it describes today's systems.

It largely does not. A 2026 replication ran the same test on current frontier models and found that the collapse reproduces only when the “irrelevant” clauses are left unaudited. Once they are filtered to genuinely irrelevant statements, the accuracy drop is statistically indistinguishable from zero (Sturgeon, 2026). The most plausible reading is that the models were making a reasonable inference — that information a questioner bothered to include might be relevant — rather than failing to reason.

The strongest published rebuttal to the collapse finding is, on inspection, an argument for exactly the fix we advocate. When models are asked to emit a compact procedure — a generating function — instead of an enumerated answer, they recover on instances previously scored as failures (Lawsen, 2025). That is the point: stop asking the model to grind out the answer in-context, have it produce the formal object, and let an engine execute it.

4. The fix: route to the mechanism that fits

If the blind spots come from a mismatch between the task's object and the model's prior, the remedy is to detect the mismatch and route the operation to a mechanism whose bias is the right shape. Four destinations cover the cases.

Three of these keep the language model in the role it is actually good at — understanding the request and explaining the result — while the exact work happens elsewhere. The division of labour is the whole point. A specialized model earns its place only where the object is inferable-but-not-computable; wherever the object is exact, a tool or a solver holds it; and wherever the model produces something checkable, a checker confirms it. Often that checker is a piece of running code, not another model, because a decidable rule deserves a deterministic test.

5. Three worked examples

Exact figures. “Multiply these two 15-digit identifiers and apply a 4.5% adjustment.” The model writes the expression; an arbitrary-precision arithmetic tool computes it; the model reports the result. The carry problem disappears because the model never performs the arithmetic.

Scheduling under constraints. “Schedule 12 crews across 30 shifts under these 40 constraints, with no back-to-back nights and at least two crews per shift.” This is past the depth where a model reasons reliably, and more effort will not help. The model translates the problem into a formal constraint model; a solver searches and returns a certified schedule or proves none exists; the model explains it.

A wide, messy spreadsheet. “Reconcile these two 60-column sheets and flag rows where 2024 sales rose but expenses fell.” The grid is held in a store and queried, so the model never has to track columns in a flattened line. Where the layout itself carries meaning — merged headers on a scanned sheet — the structure is read visually and any numbers that reading produces are checked before they are used.

6. What remains hard

Honesty about the boundary is part of the method. Some things have no reliable mechanism today, and the right response is to say so rather than to route them somewhere that only appears to answer. Reading a complex visual table with a vision model, for instance, produces model-origin numbers that must still be checked; until a suitable checked path is in place, such outputs are treated as provisional. And a question with no fact of the matter — a genuinely subjective or normative one — is answered as a set of positions, not as a result. Naming these cases is what keeps the confident-but-wrong failure from reappearing at the edge.